In the field of data processing, and especially in association with mass spectrometry, it is well known that the volume of data collected can be relatively large when compared to conventional data storage devices. It is also well-known that a typical spectrum includes a high percentage of data attributable to useless information.
For purposes of illustration of the volume of data which may be collected using conventional methods, a typical spectrum having 262,144 (256K) points and a height of three bytes is examined. In this example, one complete spectrum requires 768 kB of storage space. At a data acquisition rate of ten spectra per second, data flow is 7.5 MB/sec. However, it is well known that data acquisition rates may be higher than that illustrated. Comparing the data acquisition rate of 7.5 MB/sec to a standard, continuous, I/O rate for a conventional hard drive of 5 MB/sec, it is clear that data compression is required in order to sustain data transfer at the rate of ten (10) spectra/second for a period of thirty (30) minutes. A compression ratio of 2:1 would allow continuous transfer of the compressed data to a hard disk.
In the above example, for a data acquisition period of 30 minutes, a total storage space of approximately 14 GB is required for uncompressed data. For currently available hard disk drives of 8 GB capacity, a compression ratio of 2:1 will allow storage of the compressed data. Compression ratios greater than 2:1 are useful for compressing the data into an even smaller storage capacity.
In U.S. Pat. No. 5,592,402, Beebe, et al., teach a method for comparing spectra typically acquired from a chromatograph run on a production sample to the spectral features from a calibration standard for the purpose of detecting a maverick spectrum or for detecting a sample whose composition lies outside tolerable limits. Beebe, et al., describe methods for separating a spectrum into components comprising peaks, background, and noise. These methods are well-known in the art as described by Coldwell, Robert L. and Gary J. Bamford, The Theory and Operation of Spectral Analysis Using ROBFIT, American Institute of Physics, New York, 1991; Jenkins, Ron, R. W. Gould, and Dale Gedcke, Quantitative X-Ray Spectrometry, Chapter 6, Marcel Dekker, New York, 1981; Bevington, Phillip R., and D. Keith Robinson, Data Reduction and Error Analysis for the Physical Sciences, McGraw-Hill, New York, 1969; and Mariscotti, M. A., Nucl. Instrum. & Methods, 50, page 309 (1967).
U.S. Pat. No. 5,428,357 issued to D. Haab, et al., discloses a method for compressing data to achieve high speed data acquisition. The data compression schemes disclosed by Haab, et al., involve generating a first difference spectrum and coding sequentially repeated numbers by the common value and a number that specifies the number of times that number is repeated. However, Haab, et al., do not disclose a method for determining which portions of the data are not useful and which, therefore, may be discarded. Accordingly, unwanted data, though potentially compressed, is maintained.
U.S. Pat. No. 4,490,806 issued to C. G. Enke, et al., teaches yet another method for data compression associated with spectral analysis. In the method disclosed by Enke, et al., a fixed threshold is assigned to the data, with any data above the threshold being kept as peak data and any data below the threshold being discarded as background noise. However, there is no provision for continuously determining the threshold based upon the current values of noise in the background data. Because the threshold is fixed, there is no provision for accounting for lower peaks falling within regions of the spectrum having lower background noise levels.
The prior art of record does not disclose a method for adaptively filtering spectrometry data for compression of the same in order to eliminate the transfer and storage of unnecessary data such as background and noise.
Accordingly, it is an object of this invention to provide a means for filtering background and noise data from a spectrum in order to minimize the data transfer rate and storage requirements for the spectrum.
It is also an object of the present invention to provide such a means whereby peak data and data immediately preceding and immediately following the peak is kept while remaining data is discarded.
Another object of the present invention is to provide such a means for filtering and compressing data whereby the scatter associated with the spectrum is determined using a background noise estimate which is adaptive to ensure an accurate estimate of the scatter, thereby improving the accuracy in separating peaks from background to improve the detection limits for peaks whose amplitudes are close to the noise level in the background.
Still another object of the present invention is to provide such a means for filtering and compressing data whereby any lag in the startup of the data collection at the beginning of a spectrum is accounted for.
Further, it is an object of the present invention to provide a means whereby an increase or decrease in the background is recognized as such, without being misinterpreted as being a wide peak.
Yet another object of the present invention is to provide such a filtering and compression device whereby the sensitivity thereof is adjusted in order to compensate for sparse data.
Another object of the present invention is to provide a periodic sampling of the discarded background in order to preserve a minimal record of the shape of the background.